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ABSTRACT 


The increasing complexity of our every day jobs requires us to pursue flexible and 
more adaptive technologies with which to respond to our professional requirements. One 
such method is an expert system. This computer software "tool" is one means to augment 
and streamline ones professional decision making process. The expert system can be used 
as a means to assist new or inexperienced personnel to make informed decisions about 
their jobs. It can also assist in the decision making process when the technical expert is not 
present. Due to the fast paced, rapidly changing nature of computer software develop- 
ment, the need exists for a specific methodology to direct the development and acquisition 
of this technology within the Department of Defense (DoD). 

This study will provide an objective summary and analysis of specific contractual 
considerations that need to be addressed with regard to the acquisition of an expert 
system. A selected review of DoD and industry responses to personal interviews, 
conference presentations and published papers, served as the basis for discussion of the 


problems and issues in this arena. 
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I. INTRODUCTION 


A. GENERAL 

The increasing complexity of our every day jobs requires us to pursue flexible and 
adaptive technologies with which to respond to our professional requirements. One such 
technology is an expert system, often referred to as a knowledge based system (KBS). 
An expert system is essentially a computer software application that assists, or in a 
number of situations makes decisions based on the problem solving capabilities of an 
expert. It actually manipulates knowledge. [Ref. 1:p. 24] This computer software "tool" 
is one means to augment and streamline one’s professional decision making process. The 
expert system can be used as a means to assist new or inexperienced people to make 
informed decisions about their jobs. It can also assist in the decision making process 
when the technical expert is not present. Due to the fast paced, rapidly changing nature 
of computer software development, the need exists for a specific contracting methodology 
for use in the development and acquisition of this technology within the Department of 
Defense (DoD). 

This study will provide an objective summary and initial analysis of specific 


contractual considerations that need to be addressed with regard to the acquisition of an 


expert system. 





B. AREA OF RESEARCH AND OBJECTIVES 

This research effort studies the varied and peculiar contracting issues regarding the 
acquisition of expert systems. For the purposes of this research, acquisition refers to the 
procurement of the total system. The total system or "life-cycle" approach encompasses 
the entire procurement action to include system design, specifications, solicitation, 
contract award, and system maintenance. The object of this research is to review current 
practices, and to explore options available to program managers and contracting officers, 
that may improve the success of the development and acquisition of expert systems. 
C. RESEARCH QUESTIONS 

1. Primary Question 


What unique contractual considerations are involved in procuring an expert 
system for use in the DoD? 


2. Subsidiary Questions 


a. Of the expert systems currently acquired by the DoD, how satisfied 
are the agencies with these systems? 


b. What were the contractual problems associated with the acquisition 
of these systems? 


c. What are the observations of industry regarding satisfaction in the 
application and performance of expert systems, and any special considerations involved 
in the acquisition of these systems? 

d. What are the specific issues regarding contract type, contract 
administration, research and development (R&D), prototype development, production, and 
maintenance as they apply to the acquisition of expert systems? 

D. SCOPE 


This thesis is limited to the survey and analysis of specific issues associated with 


the acquisition of expert systems. It will attempt to identify areas peculiar to the 


development, acquisition, and contract management of an expert system. It provides an 
objective summary and analysis of specific contractual considerations that need to be 
addressed with regard to the development and acquisition of such a system. 
E. METHODOLOGY 
The research methodology used for this study included the topics discussed below. 
1. Literature Search 
A literature review was conducted in order to gain familiarity and insight 
into the area of expert systems, and to determine if there is, in fact, a need to identify any 
peculiarities involved in the acquisition of such a system. Initial sources of information 
for this research were various Federal and DoD documents addressing the acquisition of 
automated data processing equipment and software. Additional information was obtained 
from the Naval Postgraduate School Library, and the Defense Logistics Studies 
Information Exchange. The review consisted of both Government and non-govemment 
publications. 
2. Professional Conference Attendance 
Two professional conferences were attended in order to meet personnel 
versed in the area of expert system development, procurement, management, engineering 
and usage. The conferences attended are listed in Appendix A. 
3. Interview 
Interviews were conducted with both Government and non-government 
professionals. The interviews were obtained via personal and telephone conversations 


among people with varying degrees of experience. Expertise in the area of expert systems 











ranged from program managers and contracting personnel, to software/system engineers, 
to research, development, test, and evaluation personnel, to end users. The interviews 
served as a means of supplementing and augmenting information obtained through the 
literature review and attendance at the professional conferences. A list of the people 
interviewed is contained in Appendix B. 
F. LIMITATIONS 

Although expert systems have been used for several years, there is virtually no 
concise documented guidance on the actual acquisition of such a system. While there is 
an abundance of information regarding software acquisition, little specifically addresses 
expert systems. The majority of information has been through interviews and 
interpretation of actual contracting experiences from Government and non-Government 
sources. Every attempt has been made to ensure objectivity and focus on specific 
development and acquisition contracting issues regarding expert system acquisitions. 
G. ORGANIZATION 

Chapter II presents background information and reviews current practices regarding 
expert systems within the DoD. Additionally, the level of user satisfaction with these 
systems, as well as any contractual problems associated with these systems, are identified 
and discussed. 

Chapter III addresses current practices from industry regarding expert systems. 
The level of satisfaction with these systems, along with the contractual methodologies and 


problems associated with these systems, are identified and discussed. 














Chapter IV presents the varied contract types available as outlined in the Federal 
Acquisition Regulation (FAR). A discussion of the importance of choosing the 
appropriate contract type for the situation at hand is addressed in terms of suitability and 
need. The contracting officer’s role in this situation is also discussed. 

Chapter V draws conclusions from the information gathered in this research effort. 
Recommendations on how to improve and enhance the procurement of expert systems for 
use within the DoD arena are proffered. Finally, the Chapter closes with recommen- 


dations for future research. 





Il. BACKGROUND AND CURRENT PRACTICES IN DoD 


A. INTRODUCTION 

In order to gain the appropriate perspective, it is relevant to note how this 
particular research topic was decided upon. Following this discussion, and prior to 
addressing current practices and applications, the expert system will be defined, and 
certain identifying characteristics addressed. 

The initial focus was to consider the contracting requirements of an expert system 
titled Expert System Advisor for Aircraft Maintenance Scheduling (ESAAMS). This 
project is in the early stages of development by the faculty and students of the Naval 
Postgraduate School. Upon initial review, it became obvious that little research had been 
done in the area of expert system acquisition. For this reason, this research effort was 
altered to address the larger arena of expert system acquisition in general as opposed to 
a specific system. The broader scope of the research will be useful to expert system 
acquisitions, vice a particular application. 

One way to define an expert system is to compare it with an ordinary software 
program. According to Waterman, "the most basic difference is that expert systems 
manipulate knowledge while conventional programs manipulate data." [Ref. 1:p. 24] 
Artificial intelligence (AI) researchers have concluded that expert systems have the 
following distinguishing characteristics: 


* Expertise 
~ Symbolic Reasoning 





. Depth 
* Self-Knowledge 


An expert system must contain "expertise". It must "achieve the same levels of 
performance in the domain of interest that human experts can achieve.” [Ref. 1:p. 25] 
In addition to expert performance and skill, the system must have breadth or "robustness" 
as well. Waterman maintains that this area is one of the least developed characteristics 
in expert systems today, “but one that human experts can do easily." [Ref. 1:p. 25] 

"Symbolic reasoning" means rather than using equations or algorithmic 
mathematical computations to solve problems, an expert system does so by emphasizing 
the choice of symbols. "An expert system manipulates these symbols rather than 
performing standard mathematical computations.” [Ref. 1:p. 26] 

"Depth" refers to its effective operating capability within a narrowly defined and 
challenging domain. Expert systems work in what AI scientists call "real-world problem 
domains." [Ref. 1:p. 26] In such a domain, "the problem solver applies actual data to a 
practical problem and produces solutions that are useful in some cost-effective way." [Ref. 
I:p. 27] 

"Self-knowledge” or "metaknowledge" indicates a "knowledge about knowledge." 
[Ref. 1:p. 28] This is an inherently important characteristic of an expert system. Such 
knowledge allows an expert system to understand its own operation in addition to 
containing a built in structure that facilitates this reasoning process. This type of 
knowledge is important to expert systems for the following reasons: 


* Users tend to have more faith in the results, more confidence in the 
system. 














* System development is faster since the system is easier to debug. 


. The assumptions underlying the system’s operation are made explicit rather 
than being implicit. 


* It is easier to predict and test the effect of a change on the system 
operation. [Ref. 1:p. 29] 


As stated previously, the purpose of this research effort is to focus on the 
acquisition aspects of an expert system as opposed to the technical aspects of the system 
itself. Given the above definition and characteristics of an expert system, the following 
section will address current applications within the DoD. 

B. CURRENT PRACTICES IN DoD 

Current approaches to software design and development in DoD tend to be one of 
two widely used methods. The classical approach known as the "waterfall method" lends 
itself quite well to current Government practices. The waterfall development method 
requires preliminary definitions of requirements and detailed specifications or statement 
of work. While being specifically delineated, the rigid design specifications often serve 
to hinder or reduce the flexibility needed by the contractor in developing a software 
system. The real world application of the waterfall method tends to greatly inhibit the 
flexibility in software development. The restriction stems from the rigid and clearly 
defined sequence indicative of the waterfall model. These same restrictions hold true in 
contracting for an expert system developed using a waterfall methodology. [Ref. 2:pp. 34- 
35] 


Today, the generally accepted approach to contracting for a software system 


development is known as “rapid prototyping.” This is an iterative process which allows 








greater user-developer interface. The user provides initial guidance to the developer. 
These specifications are often ambiguous. The user remains in constant contact with the 
developer throughout the system development. Initial concepts and specifications change. 
A prototype, however, is able to be developed rapidly based upon initial guidance from 
the user. Because of the user-developer interface throughout the process, the inevitable 
changes that occur are easier to respond to by the developer, and the prototype can be 
changed and altered as necessary. The prototype serves as a working example for the 
user to see if the system really works as intended. Multiple prototype iterations are made, 
each expanding the scope of the system. The close coordination and communication 
between the ultimate end-user and the developer ensure that the final system is in fact 
what the user intended. [Ref. 2:pp. 35-37] 

While rapid prototyping seems an efficient means of developing a software system, 
it is not necessarily in line with current Government acquisition methodologies. These 
methodologies are predominantly hardware oriented. Present regulations require that 
specifications be delineated in the contract prior to contract award. The rapid prototyping 
method calls for initial "broad functional descriptions to loosely define the end product’s 
objectives" vice the work to be performed. [Ref. 2:p. 37] Given the disparity between 
what method is most efficient, and what the system calls for, Government acquisition 
personnel are attempting to fit the rapid prototyping method into the present hardware 
system. At the same time, they are looking at ways to make the development process for 
expert systems adapt to the peculiar needs of conventional software development and 


acquisition. 








So far, this discussion has focused on the methodologies used for developing and 
fielding computer software programs in the Federal Government. Because expert systems 
are software, their development and acquisition would seem to be of a similar nature. 
Although there are similarities between the accepted methods for procuring expert systems 
and common software applications, there are also differences. Before addressing the 
differences, however, it is necessary to note a few of the recognized myths and facts 
regarding expert systems projects. 

Mr. A.F. Umar Khan, Program Analysis Division, 7th Communications Group 
(United States Air Force), has done extensive research into streamlining the development 
and acquisition of expert system applications. His findings and current DoD trends in the 
development and acquisition of expert systems have been noted in several papers and 
presentations. At an Institute of Electrical and Electronics Engineers, Inc. (IEEE) 
conference on "Managing Expert System Programs and Projects" held on September 10- 
12, 1990 in Bethesda, Maryland, he addressed his current findings. He has identified 
myths and facts regarding expert system development and acquisition. They are listed in 
Table 2-1. 

Given these myths and facts about expert systems, it becomes evident that expert 
Systems development and acquisition techniques are in fact similar to conventional 
software methodologies. Mr. Khan noted that "a recurring point" in his research was that 
expert system development and acquisition projects are essentially software engineering 
efforts. By viewing these projects as software engineering efforts, “lessons learned 


building software over the past decades still apply." [Ref. 4:p. 33] He also stressed that 


10 








TABLE 2-1 


MYTHS AND FACTS 
Source: Ref. 3: 3 


MYTHS: > Expert system development should not be managed by the 
traditional management information system (MIS) department. 


* Expert system development is radically different from 
conventional software development, i.¢e., existing standards 
cannot be applied. 


* Expert system development varies too much and is dependent 
on the application, i.e., it is impossible to establish a single 
model. 

FACTS: bd Expert system development projects are software engineering 


efforts, i.e., lessons learned building software over the past de- 
cades can be applied. 


. Most peculiarities of expert system development which have 
been much publicized in the popular literature are not totally 
unique to expert system projects. 


* Failure to provide familiar, comfortable management controls 
contributes to low acceptance of expert systems. 


these "lessons learned" can also be applied to expert systems. He indicated that expert 
systems should be viewed in a similar context as conventional software. 

Prior to addressing the recommended methodologies for acquiring an expert 
system, it should be noted that there are essentially two ways to acquire a system 
regardless of system size. One method is to acquire a standard commercial "off the shelf" 


program "shell". Depending upon the application, funding constraints, and time 
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requirements, the requiring organization may opt to purchase a ready made commercial 
expert system "shell". The "shell" allows the requiring organization to tailor the expert 
system to their specific organizational needs. It provides the problem solving capabilities 
of an expert system at less cost than if the organization sought to develop its own system. 

Another option is to program the expert system from scratch. In this instance, 
additional considerations need be addressed. Along with cost, application, time and 
funding constraints, the requiring activity will have to consider the availability of trained 
MIS personnel, resident "experts", a focused and specialized application, and maintenance 
over the life of the system. 

Mr. Khan’s study primarily addresses medium to large scale expert system 
projects, and proffers a life-cycle approach to the development and acquisition process. 
His recommended approach attempts to merge expert system development and acquisition 
to the DoD life-cycle model. [Ref. 5:p. 16] He identifies the importance of this approach 
in that these systems "require more documentation and control during development than 
the smaller, well defined, highly constrained, stand-alone systems.” [Ref. 4:p. 32] He 
adds, however, that "it is preferable that small systems comply with the same life-cycle 
model as larger systems and that they adhere to similar control processes.” [Ref. 4:p. 32] 
An explanation of the recommended life-cycle approach will be given later in the Chapter. 


Cc. COMMONLY CITED PROBLEM AREAS IN EXPERT SYSTEM DEVEL- 
OPMENT AND ACQUISITION 


Throughout this research effort, there have been recurring references to certain 
areas of concem. Both in personal interviews and conference presentations, similar 


concems have been raised regarding problems associated with the acquisition and 
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development of expert systems. Similarly related concerns have been addressed in the 
papers, reports, and books reviewed in conjunction with this research. The following 
discussion presents the commonly noted areas of concern from the Federal Government 
perspective. The interview techniques used in obtaining this information were face to 
face and telephone type interviews. A discussion of industry’s concerns and conclusions 
follows in the next Chapter. 


1. Five Most Cited Problems Regarding Expert System Acquisition and 
Development In DoD 


A study on software contracting conducted by Major Henry Attanasio in 
1990 was used as a reference in helping interviewees to identify their concerns. [Ref. 2: 
p. 49] The five most commonly cited problems for expert systems development from the 
DoD personnel perspective are listed in Table 2-2. 

2. Recommended Rationale And Possible Solutions To The Problems 

Based on the interviews, two major issues were raised in linking the cited 
problems with a cause and potential solution. The two issues are: 

: Development and acquisition methodology 

a Training qualified and experienced personnel 

As previously discussed, the currently preferred method of rapid 
prototyping is one method in which currently perceived problems can be addressed. By 
using the rapid prototyping methodology, requirements issues are responded to as part of 
the development process. Additionally, concerns regarding reliability and maintainability 
are addressed as a normal part of the development process. The interface between the 


user and contractor throughout the process ensures these issues are addressed. [Ref. 6] 
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TABLE 2-2 


COMMONLY CITED PROBLEMS FROM DOD 
Source: Ref. 2: 49 


. Lack of qualified Government technical personnel 
bg Unclear user requirements 


* Lack of understanding regarding expert system design and 
development 


" Difficulty. in measuring reliability and maintainability of expert 
systems 


- Too many changes to initial requirements 


In order for the rapid prototyping method to work, there exists a need for 
qualified and trained people. The user has to be familiar with software terminology and 
the technical aspects of computer systems applications, as well as willing to leam about 
expert systems. The user should not only be well versed in determining the specifications 
or statement of work, but must decide on who will maintain the system. The interviewees 
agreed that the more knowledgeable the user is, the easier the development, acquisition, 
and implementation will be. 

D. CONTRACTING ISSUES REGARDING EXPERT SYSTEMS 
Specifically addressing expert systems acquisition and development in terms of 
contracting issues is of paramount importance. The contract is perhaps the single most 


important document in the acquisition process. It addresses the issues of cost, schedule, 
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technical performance, and maintenance support, not to mention the question of risk, legal 
requirements, and administration. The contract serves as the guiding document by which 
the entire acquisition is conducted. It is not just a legal basis for performance, but a 
vehicle through which responsibilities are delineated for both the contractor and the 
Govemment. This section will address the issues stated above. The contract type, which 
plays an important role in assigning the level of risk to either party, is discussed in 
Chapter IV. 
1. Cost 

Cost plays an important role in that it determines what the total price of 
the system will be. Price is the sum of cost plus a fair and reasonable profit. It is easy 
to see the importance of cost because profit is largely based on it. 

For expert systems, cost can be a difficult issue to understand. Cihan H. 
Dagli of the University of Missouri-Rolla, identified two kinds of expert system costs: 
"One Time" and "Ongoing". The following "one time” costs were identified: 
Software shell purchase 
Software development 
Other software purchase 
Hardware lease or purchase 
Communication equipment 


Office space and furnishings 
Training and documentation 


e& © © &© & & 


In addition to the "one time" costs, the following "ongoing" or “recurring” 


costs were also identified: 


* Operating personnel 


* Communication lines 
* Hardware maintenance 
15 














. Software upgrades 
* Office space and utilities [Ref. 7:p. 119] 


While not specifically stated in the above terms, most of the DoD people 
interviewed were aware of the above costs. The key areas that DoD people seem to focus 
on regarding cost of a system were software development, training and education, 
operating personnel, and software upgrades. Regardless of the costs, and the fact that AI 
technology is a new and rapidly changing field, an understanding of costs associated with 
expert systems acquisition by contracting personnel can help reduce these costs to the 
Government. 

By understanding the costs associated with expert system development and 
acquisition, the Government can realistically deal with contractors. According to Mr. 
Khan, by following the iterative rapid prototyping approach for development and 
acquisition, many costs can be significantly reduced. Inherent in the rapid prototyping 
methodology are management controls. Such controls govern the user-developer interface 
in addition to any design, development, and change actions. Because of the degree of 
user involvement throughout the process, the user will likely be satisfied with the end 
product at final delivery. The likelihood of a large number of changes and modifications 
after final delivery is lower in this process as opposed to one which requires rigid 
specifications up-front. [Ref. 6] 

2. Schedule 

The primary discussion of schedule as a part of the contract terms was in 

relation to the rapid prototyping methodology for expert system development and 


acquisition. Rapid prototyping combined with the life cycle approach recommended by 


16 











Mr. Khan seemed to be the widely accepted vehicle for controlling the development as 
well as delivery schedule. 

Mr. Khan noted the varying degrees of schedule risk associated with 
different stages of the system development process. The degree of schedule risk tends 
to decrease as the process changes from concept development, to design, to prototype 
development. As will be discussed in Chapter IV, when combined with an incentive type 
contract, the Government tends to have greater control over the contractor. Similarly, the 
contractor is incentivized to deliver on time. [Ref. 6] 

3. Technical Performance 

Performance of the system, as described in the contract is obviously a key 
ingredient of the overall procurement. While the specific performance requirements will 
vary depending on the system usage, the unique performance requirements for a specific 
application should be specified in the contract. In accordance with the currently accepted 
rapid prototyping methodology, preliminary performance is based upon "broad functional 
descriptions that loosely define the end product’s objectives." [Ref. 2:p. 37] By allowing 
the contractor a certain degree of latitude in developing an expert system, the system 
prototype can be viewed as a test bed for the user. The prototype allows the user to 
actually see if the system will meet given requirements. The prototype also allows the 
user to specify further requirements, and the contractor to implement and respond to 
suggested changes. 

Mr. Jay Griesser, Technical Application Branch, Navy Finance Center, 


Cleveland, Ohio, specifically addressed the issue of technical performance. While the 
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finance center has conducted its expert system development "in house”, Mr. Griesser 
stressed the importance of technical performance. He indicated that the users were able 
to remain in continuous contact with the knowledge engineers and system developers 
throughout the development process. In a similar vein as the user-contractor relationship 
in the rapid prototyping methodology, the ability for the user to evaluate the system from 
a technical performance aspect is not only necessary, but "should be required.” [Ref. 8] 
This system evaluation by the user is inherent within the rapid prototyping process. By 
requiring the evaluation as part of the terms and conditions of the contract, and effective 
management control by the program manager and contracting officer, technical 
performance can be reasonably assured. 
4. Maintenance Of Expert Systems 

While largely ignored in literature as well as professional conferences, and 
paper presentations, the issue of expert system maintenance is important. Maintaining the 
system, or undertaking "a set of software engineering activities that occur after software 
has been delivered to the customer and put into operation", is one way of looking at the 
maintenance issue. [Ref. 9:p. 1] Current estimates indicate software maintenance entails 
as much as seventy percent of the overall software life-cycle costs for any given software 
project. [Ref. 9:p. 1] Knowledge in an expert system must be current if the system is to 
be used. It will require constant updating. The potential for expert system maintenance 
costs to be as great or even greater than software development expenses is real. 

One way of reducing or controlling maintenance costs was to "do all of the 


maintaining in house.” [Ref. 8] In doing so, an activity could save on the costs of 
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contracting out for maintenance support. Due to the nature and complexity of the varied 
applications of expert systems, the interviews indicated that in-house maintenance was the 
exception and not the norm. Training and education for experienced and qualified 
personnel will be required to conduct in-house maintenance. [Ref. 8] 

A final problem stemming from the issue of expert systems maintenance 
is cost. Not the cost of contracting for the required maintenance, but the cost incurred 
in the development and implementation stage of the acquisition process. In considering 
this issue, Professor Martin J. McCaffrey, Naval Postgraduate School, Monterey, 
California, suggests that because of the "significant costs of future maintenance, these 
issues are often ignored in development and implementation.” Program managers or 
contracting officers may not be adequately addressing this issue. [Ref. 9:p. 5] Prof. 
McCaffrey states that failure to address maintenance issues during development is a 
"significant impediment to both the growth of expert systems in the future and the life 
cycle management of these systems." [Ref. 9:p. 8] 


E. EXPERT SYSTEM DEVELOPMENT PROCESS AND THE DoD LIFE- 
CYCLE MODEL 


The following section presents the DoD life-cycle process for automated 
information systems (AIS) as outlined in DoD Directive 7920.1. The expert systems (ES) 
development process is then "mapped" onto the DoD model. [Ref. 5:p. 16] 

The various phases as outlined in the DoD model for AIS development are 
identified in Table 2-3. These phases are for the development of conventional software 


systems, and are not peculiar to the development of expert systems. 
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TABLE 2-3 


DOD MODEL FOR AIS DEVELOPMENT 
Source: Ref. 5: 16 


Needs-Justification Phase 

Milestone 0 (Decide WHAT is wanted.) 
Concepts-Development Phase 

Milestone I (Decide HOW to do it.) 
Design Phase 

Milestone II (Decide if DESIGN is OK.) 
Development Phase 

Milestone [ll (Decide if SYSTEM is OK.) 
Deployment Phase 

Milestone IV (Decide if successful.) 
Operations Phase 

Milestone V (Decide if new system is needed.) 





The development process for expert systems, as identified by Mr. Khan, is 
portrayed in Table 2-4. This process is based on the rapid prototyping methodology, and 
applies specifically to expert systems. It addresses the expert system development from 
initial design, through the prototype iterations, and on to actual deployment. 


Table 2-5 portrays the DoD model for AIS development and the proposed ES 


development model "mapped" together. 








TABLE 2-4 


PROPOSED ES DEVELOPMENT PROCESS 
Source: Ref. 5: 16 





Initiation Phase 
Concept Prototype 
Demonstration Prototype 
Testbed Prototype 
Operational Prototype 
Deployment Phase 
Post-Deployment Phase 


aa aaa 1) (5 J mm ae et 


ES PROCESS "MAPPED" ONTO THE DOD MODEL 
Source: Ref. 5: 16 





AIS ES 

Needs-Justification Phase Initiation Phase 
Milestone 0 

Concepts-Development Phase Concept Prototype 
Milestone I 

Design Phase Demonstration Prototype 

Testbed Prototype 

Milestone II 

Development Phase Operational Prototype 
Milestone II 

Deployment Phase Deployment Phase 
Milestone IV 

Operations Phase Post-Deployment Phase 
Milestone V 
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The basic premise of "fitting" the rapid prototyping model to the DoD model is 


* Work with the model currently followed throughout DoD for AIS 
procurement. 


* Ensure that rapid prototyping is the method of choice for expert system 
development and acquisition. 


Although expert systems to date tend not to be as large as the systems addressed in the 
DoD model, the expert systems model "can be mapped straightforwardly into the AIS life- 
cycle management phases of 7920.1.” (Ref. 5:p. 15] By following this recommended 
process, issues of integration, user participation, and dynamic documentation are more 
adequately addressed in the realm of the total or life-cycle approach to expert systems 
development and acquisition. 
F. SUMMARY 

This Chapter has presented an overview of the background, current practices and 
applications, and specific contracting related issues regarding expert system development 
and acquisition. While not addressing specific expert systems, this Chapter focused on 
commonly cited problems and contracting issues deemed significant to understanding 
current trends. By identifying the problems, as well as presenting pertinent contracting 
issues, the researcher is attempting to set the stage for presenting various conclusions and 


recommendations in Chapter V. 





Ill. CURRENT PRACTICES IN INDUSTRY 


A. INTRODUCTION 

In order to gain an accurate picture of current endeavors in the acquisition of 
expert systems, it is important to contrast the efforts of the Government with a look at 
industry. The previous Chapter discussed the concems from DoD representatives. This 
Chapter will present areas of concern from industry as well as particular contracting 
issues deemed important by industry professionals. Interviews were the primary source 
of findings. 


B. COMMONLY CITED PROBLEM AREAS IN EXPERT SYSTEM DEVEL- 
OPMENT AND ACQUISITION 


Similar to their DoD colleagues, industry professionals were specific and direct 
in determining and identifying problems associated with the development and acquisition 
of expert systems. The majority of comments solicited from industry representatives dealt 
with expert systems that were designed, developed, and acquired for a specific application 
as opposed to commercial "off-the-shelf" shells. 


1. Five Most Commonly Cited Problems From Industry Regarding 
Expert Systems Acquisition 


The five most commonly cited problems from industry regarding expert 
systems acquisition are identified in Table 3-1. As with the Government personnel 


interviewed, Major Attanasio’s list of problems was used as an aid. 
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TABLE 3-1 


COMMONLY CITED PROBLEMS FROM INDUSTRY 
Source: Ref. 2: 49 








. Unclear user requirements 
* Too many changes to initial requirements 
* Too many Government regulations 
* Too many Government specifications 
* Inadequate Government specifications 
2: Recommended Rationale and Possible Solutions To The Problems 


While somewhat different from the problems identified in the Government 
perspective, there were two striking similarities. The issues of unclear user requirements, 
and the number of changes to the initially stated requirements, appeared as significant 
areas of concem in both the public and private sector. These two problems were linked 
together because preliminary user requirements are often unclear. Changes are required 
further on in the expert system development and acquisition process. 

While the link between the two problems seem evident, the recommended 
solutions are not. Industry professionals feel that better training and education on the part 
of the ultimate “end-user” would aid users in identifying requirements. In a similar vein, 
industry feels that closer coordination with the user is essential to developing a system 
that will be “everything the user intended.” [Ref. 10] By using the rapid prototyping 
methodology, industry representatives feel that the "necessary" level of user-developer 


“interface” would be ensured. 











The remaining commonly identified problems refer to various issues 
regarding specifications and regulations. Most people interviewed understood that the 
Federal Government was involved in steps to streamline the acquisition process. Taking 
a hard look at the “over-abundance" of regulations is viewed as a step in the right 
direction. Although the issue of Government regulations was identified as "problematic" 
and "cumbersome", no specific regulations were identified. 

The issue of specifications is the area industry people emphasized as being 
most problematic. Although not an industry representative, Mr. Khan, who regularly 
deals with industry professionals, indicated that identifying and reviewing specifications 
is where the majority of time and effort should be spent. He emphasized the importance 
of including such information as the purpose and scope of the system, as well as a de- 
scription of the development and delivery environment. [Ref. 6] 

Specifications, as described by Major Attanasio "are complete and detailed descrip- 
tions of products which are either military in nature, or are modified commercial products 
requiring special features to satisfy military mission needs.” [Ref. 2:pp. 18-19] Given this 
description, industry people felt that the technical specifications tend to be problematic. 
It is this area where developers tend to have a lot of difficulty due to specifications not 
being complete or detailed enough. On the other end of the spectrum, contractors tended 
to feel that specifications were sometimes too restrictive in nature thereby limiting the 
development process, although no specific examples were cited. 

The question of adequate training and familiarity on the part of the Government 


in determining system specifications seemed to be a recurring theme. Mr. Jim Crossen, 











of TRW, Sunnyvale, California, stressed the importance of adequate education by 
Govemment people associated with the procurement of expert systems and software in 
general. Such education should include familiarity with basic computer hardware and 
software terminology. In addition to being familiar with the terminology, procurement 
personnel must be able to envision or conceptualize what it is they are dealing with. 
While the discussion with Mr. Crossen addressed contract types and contracting issues, 
he did indicate a concern that contracting officers "get up to speed" when dealing with 
highly technical procurements such as expert systems. [Ref. 11] In other words, they 
need to understand what exactly it is that they are contracting for. Through both fonnal 
and informal education, the requisite level of understanding and familiarity can be 
attained. 

In a similar vein, Mr. Conner, of Intellicorp, mentioned not only the importance 
of adequate technical familiarity, but also a concern over the number of Government 
specifications and standards. [Ref. 10] While not being specific, he did indicate that a 
number of regulations and standards for the management of DoD software projects are 
redundant. A review of the standards and regulations identified by Major Attanasio 
seemed to reflect this point. For example, even though DoD-STD-2167A addresses “all 
aspects of software design, development and testing", DoD STD-1467 and DoD-STD- 
1703 address software acquisition from a program manager’s and life cycle perspective. 
(Ref. 2:pp. 19-21] While being important topics to be looked at, there exists certain 
degrees of redundancy and overlap between these standards. A review of the Government 


standards and handbooks presented by Major Attanasio indicates that industry’s feeling 
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regarding Government regulations, specifications, and standards has some merit. Both 
Govemment and industry tend to feel that these areas should be looked at more closely. 

Industry representatives felt that streamlining the Government acquisition process, 
as well as emphasizing rapid prototyping for expert systems and software in general, is 
the right direction to be heading. By not only streamlining the acquisition organization, 
but the rules and regulations governing the system, a lot of time and effort can be saved 
in the Government as well as industry. Mr. Crossen directly related this streamlining to 
savings in "cost, schedule, and management" areas. [Ref. 11] 
C. CONTRACTING ISSUES REGARDING EXPERT SYSTEMS 

Of the industry professionals interviewed, Mr Crossen was by far the most 
outspoken representative. As indicated in the previous section, he specifically stressed 
the importance of "cost, schedule, and management control.” (Ref. 11] For this reason, 
and in keeping with the format followed in the previous Chapter, the contracting issues 
regarding expert systems will be broken down into the areas of cost, schedule, technical 
performance, and maintenance support. Management control will be linked directly to 
each of these areas throughout the discussion. 

1. Cost 

Cost is an important issue in any contract. In the area of expert systems 

development and acquisition, it is just as important. Based upon the "one-time" and 
“recurring " costs identified in the previous Chapter, industry representatives viewed these 
as being accurate and realistic. The industry perspective differed somewhat from the 


Goverment. The Goverment placed emphasis on costs associated with software 
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development, training and education, operating personnel, and software upgrades. The 
industry perspective, however, tended toward software development, maintenance, 
acquisition method, and contract type. The maintenance issue will be addressed in the 
appropriate section. 

While closely related, software development and acquisition methods were 
generally discussed as two separate issues, although a link was always drawn between the 
two. 

2: Schedule 

The delivery schedule is widely recognized as a significant part of the 
contracting process. Largely linked with the development method, industry representa- 
tives indicated that during the development phase, it is important to concentrate on 
“common sense and rationality.” [Ref. 11] In later stages of the system development, a 
concise and restrictive schedule can often be agreed to with the assurance that delivery 
will be on time. Industry professionals seemed to be unanimous in the belief that rapid 
prototyping combined with the DoD life cycle approach as presented by Mr. Khan is the 
preferred method. 

Similar feelings regarding the link between schedule and the development 
methodology were discovered between industry representatives and DoD people. This 
sense of agreement between the private and public sectors was not surprising in that the 
two conferences which were attended as part of this research consisted of DoD and 


industry representatives alike. 
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KF Technical Performance 

Industry representatives seemed to stress the importance of technical 
performance. Technical performance refers to not only the physical performance of the 
system, but the interface with the user and maintenance technicians. This interface is a 
direct result of how well the system meets the specifications requirements as identified 
in the contract. The performance of the proposed system was viewed as an essential 
concem for both parties involved. Because of the rapidly growing technology in the AI 
field, industry people seemed to stress the current technological advances in this area. 
Again, the iterative process of developing expert systems, or software in general, lends 
itself to being vulnerable to technology change at every phase of design, development and 
production. Because of this vulnerability, it is important to note that depending upon the 
intended application of the system, the latest technology may not necessarily be the best 
alternative. 

Mr. Francisco J. Cantu-Ortiz, of the Instituto Tecnologico y de Estudios 
Superiores de Monterrey, Monterrey, Mexico, specifically addressed the importance of 
technology transfer from the university to industry. At the Managing Expert Systems 
(MES) 90 conference, he proposed several strategic goals for the furtherance of AI 
technology. Among his list of proposed strategies are: applied research and development, 
an annual program of seminars and symposiums on expert systems, and research 
agreements with industry. [Ref. 12:p. 70] These suggested strategies stress the importance 
of technology. These and similar efforts in the U.S. indicate the rapidly changing pace 


of expert systems technology. Because of the changing environment, it becomes even 
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more important for the Government and industry to maintain a close working relationship 


or interface throughout the development process. This relationship thereby ensures 
acceptable technical performance of the final product through the iterative development 
process. 

4. Maintenance Support 

When discussing the aspects of maintenance support, industry 

representatives viewed this as a concem, but never really addressed the issue in detail. 
Most agreed with the need for some kind of post delivery maintenance such as 
consultants, knowledge engineers, in-house maintenance and/or contracted-out type 
maintenance support, however, no specific recommendations were made. In acknowledg- 
ing maintenance support problems like documentation, user requests for changes, and 
quality of the software, Mr. Ed Hoffman of Intellicorp suggested that it was a cost issue. 
He indicated that the type of maintenance effort that is established and contracted for, will 
probably be driven by cost. [Ref. 13] This comment seemed to fall in line with the point 
raised by Mr. McCaffrey in the previous Chapter that maintenance costs account for 60- 
70% of the total software resources. [Ref. 9:p. 2] 
D. SUMMARY 

This Chapter has presented an overview of the current practices and applications, 
and specific contracting related issues regarding expert system development and acquisi- 
tion in the public sector. In presenting the expert systems development and acquisition 
issues raised by representatives from industry, it is evident that there are striking 


similarities between the public and private sectors. As alluded to earlier in this Chapter, 
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one reason for the similarities between Government and industry is that the two sectors 
often combine resources through conferences, papers, and research. While not addressing 
specific expert systems, this Chapter focused on commonly cited problems and contracting 
issues industry professionals deemed significant to understanding current trends. A 
combined private and public sector discussion will be presented in a number of 


conclusions, derived from the information gathered as a result of this research, in Chapter 


V. 
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IV. CONTRACT TYPES IDENTIFIED AND DISCUSSED 


A. INTRODUCTION 

According to Major Attanasio, "the single most important document in the 
procurement of custom designed software is the contract itself." [Ref. 2:p. 26] Not only 
does the contract serve as a basis for explicitly stating what is intended, but it is also "one 
of the major pricing aids available to a buyer" due to the varied contract types available. 
(Ref. 14:p. 181] The contract is essentially a collective legal agreement between the 
buyer and seller (Government and industry), clearly delineating the responsibilities for 
both parties. 

Goverment and industry use two broad contract categories: fixed-price, and cost- 
reimbursement type contracts. The Federal Acquisition Regulation (FAR) outlines each 
contract type, as well as the various factors involved in selecting and applying them. 
[Ref. 15: Part 16] The specific contract types contained in these two categories range 
from firm-fixed-price, to a cost-plus-fixed-fee. The firm-fixed-price type contract places 
full responsibility for performance costs and profit on the contractor. The cost-plus-fixed- 
fee type contract places minimum responsibility on the contractor for performance costs, 
and the ensuing profit fee is fixed. The contract types, as listed in the FAR, are identified 
in the following sections. 

This Chapter will provide a discussion of the two categories of contract types, and 
how each contract type applies to software, and more importantly, expert systems 


acquisition. 
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B. FIXED-PRICE TYPE CONTRACTS 





According to the FAR, "fixed-price types of contracts provide for a firm price or, 
in appropriate cases, an adjustable price.” [Ref. 15: 16.201] In a fixed-price type arrange- 
ment, the contractor is required to make delivery of the goods and/or services for a given 
price, on time, and in accordance with the specifications of the contract, regardless of 
costs incurred. In doing so, the Government agrees to pay the contractor a fixed-price. 
The following discussion addresses four general categories of the fixed-price type 
contracts, as presented by Major Attanasio, as they apply to software. Their application 
to expert systems development and acquisition will be presented throughout based on 
input from Mr. Crossen and Mr. Khan. 

1. Firm-Fixed-Price (FFP) 

As previously indicated, the FFP contract presents the least amount of risk 
to the Government. The FFP contract "provides for a price that is not subject to any 
adjustment on the basis of the contractor’s cost experience in performing the contract”, 
and imposes "a minimum administrative burden upon the contracting parties." [Ref. 15: 
16.202-1] According to Mr. Crossen, a FFP contract is appropriate for commercial off- 
the-shelf type purchases, but not for expert systems in the design or development stage. 
He felt that a FFP or even a FPIF type contract would be appropriate only after the 
operational prototype expert system was developed, and the process was beginning to 


enter the deployment phase of the life cycle process. [Ref. 11] 
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2. Fixed-Price with Economic Price Adjustment [FPE] 


The FPE type contract allows for upward or downward price adjustments 
to the contract based on the following three economic factors: 

(1) Established Prices 

(2) Actual Costs of Labor or Material 

(3) Cost Indexes of Labor or Material 
In allowing for such adjustments, the Government and the contractor are seeking a certain 
level of protection from these economic factors. [Ref. 15: 16.203] While not routinely 
used for software acquisition or development, it is possible according to Major Attanasio, 
and therefore should be included. 

3. Fixed-Price-Incentive (FPIS or FPIF) 

"A fixed-price-incentive contract is a fixed price contract that provides for 
adjusting profit and establishing the final contract price by a formula based on the 
relationship of final negotiated total cost to total target cost." [Ref. 15: 16.204] 

FPI type contracts are used as a means to "incentivize" the contractor to 
control costs. If the final cost is greater than the target cost, then the contractor loses 
profit. If, however, he manages to complete the contract with the final cost being less 
than the target cost, his profit will be greater. As indicated previously, Mr. Crossen felt 
that a FPI type contract would be appropriate when the contractor is in the production 


phase, and sufficient cost or pricing data were not available to negotiate a FFP. [Ref. 11] 
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4. Firm-Fixed-Price, Level of Effort (FFP,LOE) 
A FFP,LOE type contract requires the contractor to perform a specified 

“level of effort", for a specified amount of time. The work to be performed is stated in 
general terms thereby allowing the contractor a certain degree of flexibility. The 
Government is then required to pay a fixed dollar amount to the contractor for the work 
performed. [Ref. 15: 16.207-1] Regarding software acquisition and development, the 
FFP,LOE is "typically used for feasibility type study in a research or development effort." 
[Ref. 2:p. 31] In the expert system arena, the FFP,LOE or even a CPFF,LOE was 
recommended for maintenance efforts. The FFP,LOE contract was recommended by Mr. 
Crossen for situations where the "customer", or user, would have to "stay out of the 
system." The CPFF,LOE would be appropriate for situations where the user had to "get 
into the system.” [Ref. 11] In other words, when the user can actually enter the system 
and manipulate data, there is a greater potential for requiring the maintenance technician 
to be involved in more troubleshooting than he normally would be. When the user has 
no need to enter and manipulate the system data, there is less potential for problems to 
arise as a result of outside interaction. 
Cc. COST REIMBURSEMENT TYPE CONTRACTS 

Cost type contracts are best suited for use when uncertainties exist in being able 
to accurately estimate costs. The contractor is allowed to recover payment for all "allow- 
able and allocable" costs. The FAR describes these type contracts in the following 
manner: 


These contracts establish an estimate of total cost for the purpose 
of obligating funds and establishing a ceiling that the contractor may not 
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exceed (except at its own risk) without the approval of the contracting 
officer. [Ref. 15: 16.301-1] 


The following discussion presents the various cost reimbursement type contracts, 
and applies them to the development and acquisition of expert systems software. 
1. Cost-Sharing (CS) 

Although not discussed in great detail, the cost-sharing type contract was 
mentioned as an alternative. In this type contract, the contractor receives no fee, and only 
gets reimbursed for an agreed upon portion of his allowable costs. In this instance, the 
contractor generally agrees to absorb a certain portion of his costs in expectation of 
receiving compensating benefits in the future. [Ref. 15: 16.303] While not widely used 
in expert systems development and acquisition, the possibility exists for its potential use. 
A firm that is largely in the research and development business could view this type 
contract as a stepping stone to other business ventures. [Ref. 6] 

2. Cost-Plus-Incentive-Fee (CPIF) 

The CPIF type contract is a cost-reimbursement contract that provides for 
an initially negotiated fee that is adjusted based on the total allowable costs and the total 
target costs. Initially, the target cost, target fee, maximum and minimum fee, and the 
adjustment formula are generally agreed upor. Depending on the cost-share ratio, the fee 
is adjusted upward or downward dependin;, upon whether or not the contractor completes 
the contract below or above the target cost. The key is to reach an accord with the 
contractor based on the target cost and fee adjustment formula likely to motivate him to 
manage effectively and efficiently. The CPIF is appropriate for development and test type 


programs. {Ref. 15: 16.404-1] 
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3. Cost-Plus-A ward-Fee (CPAF) 

In this cost-reimbursement type contract, the contractor is allowed an 
agreed upon base amount fixed at the inception of the contract. In addition to the base, 
an award is authorized that the contractor may or may not be eligible for. The eligibility 
is based on the Government’s evaluation of the contractor’s performance in pre- 
determined areas such as quality, timeliness, technical ingenuity, and cost-effective 
management. The general criteria for the contractor’s performance are stated in the terms 
of the contract. By monetarily rewarding the contractor for his performance in given 
areas, he is incentivized to maintain a reasonably high level of performance standards in 
order to gain the maximum reward. [Ref. 15: 16.404-2] As addressed in the previous 
section, both the CPIF and the CPAF type contracts are ideally suited for the early stages 
of the rapid prototyping process. 

4. Cost-Plus-Fixed-Fee (CPFF) 

A CPFF contract allows for the contractor to be paid a fixed fee negotiated 
and agreed upon at contract inception. While adjustments can be made based on changes 
in the work to be performed, the fee is generally fixed. In this type of cost-reimburse- 
ment arrangement, the contractor is allowed the fee due to the potential risks involved in 
the performance of work. This type of arrangement does not, however, provide a great 
deal of incentive for the contractor to control costs. Under the CPFF type contract 
afrangement, the Government assumes the majority of risk. [Ref. 15: 16.306] In the 
expert systems arena, the CPFF is one vehicle recommended for use in the research and 


development phase of the DoD life-cycle acquisition process. [Ref. 6] Because of the 
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greater risk assumed on the part of the contractor in this phase, the CPFF arrangement 
ensures a certain fee in addition to the allowable and allocable costs. [Ref. 11] 
D. CHOOSING THE APPROPRIATE ALTERNATIVE 

The contract types previously discussed are available to the Goverment and 
contractors as a means to "provide needed flexibility in acquiring the large variety and 
volume of supplies and services required by agencies.” (Ref. 15: 16.101] The selection 
of the proper contract arrangement is basically a matter for negotiation and the application 
of sound judgement. The application of sound judgement throughout every phase of the 
process is essential to a successful project. The key of contract selection is to negotiate 
a contracting arrangement, and either price or estimated cost and fee: 
..that will result in reasonable contractor risk and provide the 
contractor with the greatest incentive for efficient and economical 
performance. [Ref. 15: 16.103] 
By selecting the appropriate contracting arrangement, a reasonable level or share of the 
cost, schedule, technical, and support risk is assigned to each party. The ideal 
arrangement is one that has the Government and the contractor walking away feeling like 
they both received a fair and reasonable deal; essentially a "win-win" situation. 

Management control emerged as the key ingredient throughout the entire 
development and acquisition process. Both Government and industry identified the need 
for management involvement and control in every area of the acquisition process. 

The industry feeling is that a CPIF or a CPAF is appropriate for the concept, 
definition/design, and development stages of the DoD life-cycle management process for 


automated information systems. These stages of the rapid prototyping process are the 
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times when the contractor is likely to incur unanticipated costs. While the iterative 


process entailed in rapid prototyping tends to significantly reduce overall costs, any 
developmental stage will have unforseen expenses. According to Mr. Khan, the iterative 
process tends to reduce costs by foregoing a great number of the changes and 
modifications that software systems developed under non-iterative methodologies tend to 
experience. For this reason, and to reward the contractor for efficiency and effectiveness, 
the CPIF and the CPAF are recommended. [Ref. 6] 
E. SUMMARY 

By presenting the various contract arrangements outlined in the FAR, this Chapter 
is meant to provide a basic understanding of the primary contract types available. In 
applying the contracting arrangement to the development and acquisition of expert 
systems, the popular opinion of Government and industry is to award a cost type contract 
in the early stages of research, concept definition, design, and development. Whatever 
the contracting arrangement, the key is to ensure appropriate allocation of risk, a quality 


product or service for the Government, and a fair and reasonable profit for the contractor. 
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V. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

This research has lead to specific conclusions regarding the development and 
acquisition of expert systems. The following conclusions will be addressed individually 
and then followed in the next section by recommendations. 

The first point, and possibly the most significant, is the question of what an expert 
system really is. Within the computer software community there exist areas of specializa- 
tion. One such area is artificial intelligence or AI. Expert systems fall within the 
purview of AI. Within the AI community, there are those who believe that expert 
systems are unique in and of themselves, and should be treated differently vis-a-vis 
typical computer software applications. The research indicated that expert systems should 
in fact be treated as software, and that similar approaches be taken in the areas of 
research, design, and development. The development and acquisition of these software 
systems is addressed in the DoD Directive 7920.1 regarding life-cycle management of 
automated information systems. [Ref. 5:p. 16] Program managers and contracting officers 
have to remember that they are essentially dealing with software issues. 

Another point brought about as a result of this research is the methodology used 
in the development of expert systems. The traditional or "waterfall" approach does not 
fit well with the peculiarities of expert systems development. It requires firm specifica- 
tions or statements of work early on in the development process. Because of the iterative, 


or growing and re-defining type of developmental approach in expert system development, 
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the waterfall development approach does not work for developing expert systems. The 
need for up-front specifications restricts the developer in crafting a system that will 
properly meet the needs of the user. DoD documentation standards such as DoD Standard 
7935.1, Automated Data Systems (ADS) Documentation Standards (April 1984), tend to 
emphasize hardware vice software requirements. Because of the different characteristics 
of expert systems as opposed to conventional software, ie., expert systems attempt to 
model human thought processes, and the emphasis on hardware, the developer cannot 
effectively document the expert system application as it would appear to the user. [Ref. 
4:p. 33] The waterfall approach essentially removes the user from the development 
process, thereby restricting the developer from being able to meet the specific user needs. 
{Ref. 16: 88,102] 

Documentation of the expert system development process is important, however, 
anc should be required throughout the process. A recommended approach is that of a 
"Progress Notebook" that would entail a "historic trace" of the entire expert system 
development process. The "Progress Notebook" would tend to "fill in the gaps left by the 
standard documents." [Ref. 4:p. 38] 

The popular and preferred method of expert system development is known as the 
"rapid prototyping" approach. This method allows for the user to remain closely involved 
throughout the development process. The contractor develops a prototype system that is 
tested against the user’s requirements. Because the initial system is defined in “broad 
functional terms" as identified in Chapter II, the contractor is afforded greater flexibility 


in meeting the user’s requirements. [Ref. 2:p. 37] Such descriptions could be in terms 
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of what the system is expected to do, as opposed to looks and language requirements. 
By describing the system in functional as opposed to design type terms, the developer is 
afforded the latitude to adequately respond to the users needs throughout the iterative 
process. If described in specific design type terminology, the developer is then restricted 
in developing the proposed system. Specific issues of integration, the continuous need 
for user and domain expert system participation, and the need for dynamic documentation 
throughout the process are easily managed in the context of the rapid prototyping model. 
[Ref. 5:p. 15] 

While varying somewhat in the identification of specific problem areas, industry 
and Govemment seem to be in general agreement regarding questions conceming contract 
type, cost, schedule, technical performance, and maintenance. Continued Goverment and 
industry interaction through conferences, publication of professional papers, and 
communication, will ensure these concems and problem areas will continue to be 
addressed and potential solutions attained. 

B. RECOMMENDATIONS 
As a result of this research, the following recommendations are made: 


1. Continuous Correlation Between Expert Systems and Conventional 
Software 


Although expert systems are a unique application of a software system, 
they are in fact software. [Ref. 4:p. 33] This research has shown that conventional 
software, and expert systems development and acquisition methodologies, are essentially 
the same. Issues of concem in the software development field are virtually mirrored in 


the expert systems arena. Issues such as requirements changes, specifications, technical 
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performance, cost and management control ring clear in both fields. As indicated 
previously in Chapter IV, the question of software maintenance is not only a concem, but 
significantly impacts life-cycle cost control in each phase of the process. Because of the 
similarities brought forth in this research effort, it would stand to reason for the two fields 
to maintain open lines of communication. Industry and Government recognition of these 
issues in software development and acquisition projects is essential. 
2. Use of the Rapid Prototyping Methodology 

In contracting for the development and acquisition of expert systems, the 
iterative rapid prototyping process should be used. Currently the preferred method 
throughout industry and the Federal Government, the rapid prototyping process facilitates 
the necessary management control in developing and acquiring expert systems. Rapid 
prototyping additionally ensures that the needs of the user will adequately be met. A 
common theme throughout this research has been the link between the various problems 
encountered in developing and acquiring an expert system, and the process used. Both 
industry and Government professionals tend to agree that application of the rapid 
prototyping method can significantly reduce the problems currently encountered. Con- 
cems with requirements changes, specifications, and user-developer interaction are 
addressed throughout this process. The rapid prototyping methodology allows for 
coordination of user requirements throughout the process. Because cvordination, 
communication, and management control are inherent to the rapid prototyping process, 


the Government and contractor reap the benefits of schedule and cost control measures. 


43 





3. Education and Training of Users, Contracting Officers, and Program 
Managers 


Viewing expert systems from a software perspective, and adequately 
applying the management techniques involved in the rapid prototyping methodology, can 
only be brought about through education and training. From a managerial viewpoint, 
knowledge and understanding of the proposed system is just a small part of the overall 
understanding required. In order to properly apply judgement, and make informed 
decisions about the program or acquisition, the manager and contracting officer have to 
understand the industry as well as the proposed system. From a user and developer 
perspective, a thorough knowledge and understanding of the system, system operating 
environment, applications, and alternatives will greatly benefit the overall process. 
Through educational courses and professional conferences and journals, people involved 
at each level of the development and acquisition process will be better able to respond 
in this dynamic environment. 


4. Continuous Revision and Updating of Government Regulations and 
Specifications 


Current trends in both the Federal Government and industry reflect concern 
for revising and updating acquisition regulations and Government specifications. In 
offering the views of the Aerospace Industries Association (AIA) regarding the Secretary 
of Defense’s "Defense Management Report" of July 1989, Mr. Don Fuqua, President of 
AIA, recommended that "industry should be required to participate in the initial 
development of future regulatory changes.” (Ref. 17:pp. 1-1, 2-11] This recommendation 


falls directly in line with the sentiment from both industry and Government interviewees; 











that not only should acquisition streamlining continue, but the Government and industry 
should work together. By working together, issues regarding the abundance of regulations 
and specifications, and contracting arrangements best suited for the situation can be 
addressed and responded to. As indicated previously in this paper, communication 
between Government and industry is not just a good idea, but essential to the effective 
and efficient application of the development and acquisition process. 

Cc. ANSWERS TO RESEARCH QUESTIONS 


1. What unique contractual considerations are involved in procuring an 
expert system for use in the DoD? 


As indicated throughout this research, there are few "unique" considerations 
regarding the acquisition of expert systems. On the contrary, it is important to note that 
expert systems should be viewed in the purview of software development and acquisition. 
The primary difference between expert systems and conventional software, ..s discussed 
previously, is that expert systems "manipulate knowledge" vice "data." [Ref. l:p. 24] 
Expert systems should be acquired based on the factors of need, scope, application, and 
estimated cost of the specific project at hand. Questions regarding whether to purchase 
a commercial shell, or develop the system from scratch, are a function of these factors. 
While the results of this research stress the use of rapid prototyping for expert system 
development and acquisition as opposed to the waterfall method, further research could 
be conducted to determine other feasible variations or altematives. Other concems and 
problems identified as a result of this research virtually mirrored those brought up in 


previous research in software development. 
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2. Of the expert systems currently acquired by the DoD, how satisfied 
are the agencies with these systems? 


The people interviewed throughout this research effort indicated satisfaction 
with the different expert systems at their disposal. Because of the abundance of informa- 
tion regarding other significant problems in the expert system development and 
acquisition arena, the actual systems themselves were not addressed. The interviewees 
reflected satisfaction with the actual systems, however, they were specific in identifying 
problem areas as were discussed in Chapters I and II. 


3. What were the contractual problems associated with the acquisition of 
these systems? 


From the Government perspective, the five most commonly identified 
problem areas in expert systems development and acquisition were identified in Table 2-2. 
The primary recommendations to remedy these problems, or at least lessen their impact 
are: 

- Use of rapid prototyping development and acquisition methodology 

a Training qualified and experienced personnel 
By applying the rapid prototyping methodology to the development and acquisition of 
expert systems, user and developer interaction can be managed. By ensuring early and 
continuous user interaction throughout the process, requirements can be responded to, and 
adapted, thereby limiting the level of confusion over what the user actually wants and 
needs. 

As addressed in the recommendations section, education and training of the 


personnel involved in the development and acquisition of expert systems is essential. 
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Improved management, communication, and coordination can be assured through ensuring 
that these people have an adequate understanding of the system, system environment, 
acquisition methodology, and available alternatives. 

4. What are the observations of industry (both user and producer) 
regarding satisfaction in the application and performance of expert 
systems, and any special considerations involved in the acquisition of 
these systems? 

While the problem areas identified by industry varied somewhat from those 
of the Government, their major concems were essentially the same. Additionally, industry 
expressed satisfaction with the general application of currently used expert systems. The 
predominant concerns of industry regarding expert systems development and acquisition 
were presented in Table 3-1. 

Industry sentiment seems to be in line with the Government’s in that 
unclear requirements, as well as requirements changes, often lead to cost and schedule 
overruns. Industry representatives, like their Government colleagues, felt that both 
education and use of the rapid prototyping development and acquisition methodology 
would significantly enhance the overall process. 

In discussing the issues of Government regulations and specifications, the 
words “burdensome”, "confusing", "over-abundance", and "ambiguous" often came up. 
Industry professionals agreed that the Government needed to pay particular attention to 
regulations and specifications. By continually reviewing these, and even including 


industry in the process, the overall system could incur substantial savings in the areas of 


cost, technical performance, schedule, ard maintenance support. 
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5. What are the specific issues regarding contract type, contract 
administration, research and development (R&D), prototype develop- 
ment, production, and maintenance of expert systems? 

The primary contracting issues were broken down into the areas of cost, 
schedule, technical performance, maintenance support, and contract type. Contract 
administration, while being an area of concem, was too broad in scope to include in this 
research effort. For this reason, contract administration issues such as warranties, contract 
changes, contract interpretation, disputes, and legal implications, are recommended for 
future research efforts. 

Each of the areas identified above is linked to management and 
methodology. The methodology issue is continually referred to as being the key to a 
program manager’s or contracting officer’s ability to adequately handle the situation. 
Proper planning, and appropriate acknowledgement of the contracting issues identified 
above, throughout every phase of the development and acquisition process, is one means 
to reasonably ensure success. 

D. RECOMMENDATIONS FOR FUTURE STUDY 

1. Contract Administration Issues 

Examine pertinent contract administration issues such as warranties, 
disputes, contract interpretation, contract changes, adjustments, and legal questions from 
an expert system acquisition perspective. As previously indicated, contract administration 
is definitely an area of concem in the overall development and acquisition process. A 


review of the various studies, and literature on these topics for conventional software 


acquisition can be directly related to expert systems. An interesting question is that of 








liability and risk associated with an expert system, i.e., responsibility for inaccurate 


decisions or results based on the information provided by an expert system. Issues of 
legal responsibility may or may not be addressed in the warranty. If addressed, then how 
are the issues enforced? The answer may lie in the realm of contract interpretation, or 
clearly delineating within the context of the appropriate contract clauses what is intended 
by the Government and the contractor. 


2. Formalized Education and Training in Software Development and Ac- 
quisition 


Examine the current system of education and training in the area of 
software development and acquisition. Compare the current system with various 
alternative approaches such as formal training for procurement personnel regarding 
software and hardware terminology and applications. The focus of such training would 
be in the area of expert system design, development, and production techniques as they 
apply to expert systems. Alternatives can be obtained through interviews with Govern- 
ment and industry professionals. As identified throughout this research effort, the need 
for formal education and training at every level of the software development and 
acquisition process is essential. 


3. Review Development and Acquisition Methodologies for Specific 
Expert Systems Applications 


Analyze and compare various expert systems applications from a 
development and acquisition perspective. Through in-depth analysis of selected expert 
systems projects, an accurate assessment of the development methodologies used can then 


be made. By studying actual expert systems projects, the various methodologies can be 
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assessed as to their effectiveness and efficiency. This type of research effort would 


provide professionals in the field of expert systems development and acquisition with the 
appropriate understanding of the best suited methodology to apply. 

4. Review of Government Regulations and Specifications 

Examine current Government regulations, specifications, and standards for 

appropriate application to expert systems development and acquisition. By identifying 
those Government regulations, specifications, and standards that are ambiguous or 
repetitive, measures can then be made to clarify, update, or possibly delete applicable 
items. A joint project including Government and industry representatives could be 
initiated as a means to potentially identify possible issues of concem. Government 
standards such as those identified in Chapter III may be able to be condensed, combined, 
updated, or possibly even deleted. By working with industry and Government on this 
project, the concerns regarding redundancy and overlap can be addressed. 
E. SUMMARY 

As a result of this research, certain key areas of concern from the Government and 
industry perspective were brought to light. Education and training, redundant Govemment 
regulations, specifications, and standards, and development methodology, were frequently 
discussed. Both industry and Govemment personnel tended to agree that these areas must 
be addressed from user to program manager to contractor. 

Understanding that expert systems are software, and that they should be developed 
and acquired in a similar fashion is a paramount concem. By understanding this 


correlation, use of the rapid prototyping methodology arises as the natural vehicle for 
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acquiring an expert system. Rather than change the Government approach to software 
development, a proposed method to work within this system has been recommended in 
Table 2-5. This method allows for the iterative prototype development process required 
for expert systems, as well as the DoD requirement to follow the AIS development 
process via the milestone approach. By "mapping" the two processes together, 
efficiencies in time, schedule, and cost can be attained by ensuring an adequate level of 
user-developer interface throughout. 

The remaining two areas of concern were contract type and maintenance. Most 
interviewees agreed, as does the author, that a cost type contract for the early stages of 
expert system design and development is advantageous for industry as well as the 
Govemment. In this type arrangement, industry is rewarded for assuming a greater 
portion of the cost risk associated with early development phases. In a similar vein, as 
the development process leads to an operational prototype, and enters the deployment 
phase of production, the level of risk changes. Because of the shift in risk, the type of 
contract should also shift to reflect the change by assuming a fixed-price type of 
contracting arrangement. Whether by agreeing to one contract that will change according 
the stage of development, or by contracting for each phase of development separately, 
these types of contracting arrangements should be pursued. The decision to go with one 
contract or a series of contracts depends primarily on time, cost, and competition. 

While being an important issue of concer, the area of maintenance is often 
overlooked. Because of the large proportion of cost involved in maintenance throughout 


the life-cycle of the system, maintenance should be required to be addressed and planned 
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for from the earliest stages of concept design and development. By starting early on in 
the rapid prototyping process, the user and developer can begin to respond to the 
questions of contractor maintenance, in-house maintenance, or a combination of the two. 
The need for maintenance may require that not only a clause be added to the contract 
addressing this issue, but that the proposed "mapping" of the DoD AIS model and the ES 
life-cycle model be changed to reflect this need throughout each stage of development. 

Finally, as discussed in the previous section, it is important to continue addressing 
the areas of contract administration, education and training, development and acquisition 
methodologies, and Government regulations, specifications, and standards. The rapidly 


changing and growing field of artificial intelligence will continue to raise new concerns 


over these areas. 
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